This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of 
the original documents submitted by the applicant. 

Defects in the images may include (but are not limited to): 

• BLACK BORDERS 

• TEXT CUT OFF AT TOP, BOTTOM OR SIDES 

• FADED TEXT 

• ILLEGIBLE TEXT 

• SKEWED/SLANTED IMAGES 

• COLORED PHOTOS 

• BLACK OR VERY BLACK AND WHITE DARK PHOTOS 

• GRAY SCALE DOCUMENTS 

IMAGES ARE BEST AVAILABLE COPY. 



As rescanning documents will not correct images, 
please do not report the images to the 
Image Problem Mailbox. 



.-3 



(19) 



J 



Europaisch s Pat ntamt 
European Patent Off! e 
Offic europe nds brevets 




(12) 



(43) Date of publication: 

08.08.2001 Bulletin 2001/32 



(11) EP 1 122 931 A2 

EUROPEAN PATENT APPLICATION 

(51) mtci7: H04L 29/06 



(21) Application number: 01 1 02060.9 

(22) Date of filing: 30.01 .2001 



(84) Designated Contracting States: 

AT BE CH CY DE OK ES Fl FR GB GR IE IT LI LU 
MC NL PT SE TR 

Designated Extension States: 


(72) Inventor: Ishll, Atsushl 

Vancouver, Washington 98665 (US) 

(74) Representative: mOLLER & HOFFMANN 


AL LT LV MK RO SI 


Patentanwalte 


(30) Priority: 07.02.2000 US 499701 


Innere Wiener Strasse 17 


81667 MOnchen (DE) 


(71) Applicant: S H A RP KA B U S H I Kl KAISHA 




Osaka 545-8522 (JP) 





(54) Real-time media content synchronization and transmission in packet network apparatus and 
method 



(57) A system for transmitting and synchronizing re- 
al-time multimedia content includes a multimedia server 
(62) for generating a multimedia packet (64) having mul- 
timedia audio/visual information therein in a play se- 
quence, and for transmitting the multimedia packet; a 
packet-based communication network (66) connected 
to the multimedia server (62) for receiving the multime- 
dia packets therefrom; the network having multiple rout- 
ers (68, 70) therein to route the multimedia packets to 
plural destinations; means for inserting total delay infor- 
mation (TDI) into the multimedia packets, wherein the 
TDI includes total end-to-end delay (TED) and cumula- 
tive network delay (CND); and plural multimedia receiv- 
ers (80, 82, 84) for receiving the multimedia packets 
having TDI therein, wherein each multimedia receiver 
(80, 82, 84) includes a buffer for storing received multi- 
media packets, a sequencing means for retrieving 



stored multimedia packets from the buffer and sequenc- 
ing the multimedia packets in the play sequence, and a 
playing mechanism for playing the retrieved, sequenced 
multimedia packets at time TED after the multimedia 
packet has been transmitted by the multimedia server 
(62). A method of transmitting and synchronizing real- 
time multimedia content includes loading multimedia 
audio/visual into plural multimedia packets (64); insert- 
ing total delay information (TDI) into at least one of the 
multimedia packets (64), wherein TDI includes total 
end-to-end delay (TED) and cumulative network delay 
(CND); transmitting the multimedia packets to plural 
multimedia receivers (80, 82, 84) over a packet-based 
network (66) in a play sequence; buffering the multime- 
dia packets in each multimedia receiver (80, 82, 84); re- 
calling the multimedia packets in play sequence; and 
playing the multimedia packets (64) at each receiver 
(80, 82, 84) at time TED. 
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Descriptl n 

Field of the Invention 

5 [0001] This invention relates to multimedia networks, and specifically to the synchronization of multicast distributed 
media content when played by multiple multimedia devices, e.g., monitors and audio systems. 

Background of the Invention 

10 [0002] In some situations, such as at conventions and in exhibition halls, it is required to play the same video contents 
synchronized at multiple video monitors. Generally, it is also required to synchronize audio and video between sepa- 
rately located monitors and speakers. Current real-time media transmission technology allows transferring an audio/ 
video stream over n on -guaranteed quality of service (QoS) packet networks with reasonable quality and at a lower 
cost than provided by higher quality networks. Thus, there will be a demand for playing multimedia contents, delivered 

15 to a plurality of receivers over packet network, in synchronized manner. 

[0003] Fig. 1 depicts a prior art multimedia content delivery system, generally at 10, which uses a packet network. 
A multimedia server 12 generates a stre am o f mu l timedia packets 14 t h at cu ii ta inTrrulttme d i a aud i o/vid e o con tents- 
Normally, server, or source, 12 generates multimedia packets at a constant rate (packet rate); e.g., one packet every 
20 ms. Packet size depends on the multimedia codec, which may result in fixed size or variable sized packets. The 

20 generated multimedia packets are then transferred through a packet network 16, having plural routers, or nodes, 18, 
20, 22, 24, 26, 28 and 30 therein, and are delivered to a plurality of multimedia receivers, 32, 34, 36 and 38. The 
mechanism for delivery to multiple devices is well known as "multicast." All the receivers that wish to receive multimedia 
contents from the server join a multicast group. The routers in packet network 16 are aware of the multicast group, 
and transfer received multimedia packet to multiple routes. Each receiver is equipped with a speaker and/or a video 

25 display, and each plays the contents in the received multimedia packet in real-time. The multimedia receivers may 
buffer received multimedia packets for some duration in order to absorb the variance in delay caused by the network, 
and to playback multimedia packets in right order. 

[0004] In conventional packet network, such as an IP network, packets are queued in each router. How long a packet 
stays in a queue depends on the amount of traffic that a particular router is handling. Hence, each multimedia packet 
30 delivered to each multimedia receiver has an unpredictable delay. As a result, the audio/video contents may be played 
at multiple locations without synchronization. 

[0005] As previously noted, real-time multimedia transmission technology currently allows transferring an audio/ 
video stream over non-guaranteed quality of service (QoS) packet networks with reasonable quality. As mobile Internet 
access increases, with its data transfer over bandlimited communication channels, such as wireless data networks, 
35 the technology to transmit real-time multimedia packets requiring minimum bandwidth becomes essential and of in- 
creases importance. 

[0006] A packet-based real-time media transmission system is illustrated in Fig. 2, generally at 40. A multimedia 
source, or server, 42, periodically transmits multimedia packets 44 having audio/video multimedia contents to a packet 
network 46. In packet network 46, each multimedia packet is routed by a plurality of network nodes, or routers, 48, 50, 

40 52 and 54, and delivered to a receiver 56. Each multimedia packet includes timing information that is used to set timing 
for playback at a receiver 56. The multiple routers and receivers of Fig. 1 are not shown in Fig. 2 for the sake of clarity. 
[0007] Packet network 46 causes a variable length delay in the delivery of each multimedia packet. In addition, 
network 46 does not guarantee the delivery of the multimedia packets in the same order as transmitted. Thus, a receiver 
is normally equipped with a buffer to handle delay variances and/or disorders in multimedia packet receipt. However, 

45 due to the real-time property of the contents, the receiver will discard a received multimedia packet if the delay in its 
arrival exceeds a certain value, or if the multimedia packet does not arrive by the time it is to be played. 
[0008] Fig. 3 shows an example of packets received and played back by the receiver, where each packet contains 
the timing information. Line I depicts seven multimedia packets transmitted in sequential order by server 42. Line II 
depicts the order of receipt of the multimedia packets at a receiver. Line III depicts a play order by the receiver. The 

50 figure depicts that the multimedia packet with timing information 3 is discarded due to an excessive delay. Basically, 
the routers in the prior art packet network are not aware of delays in packets transmission and receipt, nor are they 
aware of disordering of packet transmission. Thus, each router re-directs every received real-time multimedia packet 
to the next router or the receiver, even if the multimedia packet may not have any chance of being played at the receiver. 
If a physical link between some two routers, or th physical link between the rec iver and th adjacent router has 

55 limited bandwidth, the transmission of the multimedia packets which will be discarded at the receiv r possibly wastes 
substantial portion of valuable bandwidth on the link. Delay and disordering is illustrated by the variable routing of 
multimedia pack ts from router 48 through router 50, router 52 and router 54, or directly from router 48 to router 54. 
[0009] U.S. Patent No. 4,506,358, for Time stamping tor a packet switching system, to Montgomery, granted March 
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1 9, 1985, describes a method of measuring th network delay in a packet n twork by Including an information field in 
each packet that indicates th cumulative network delay, time stamping each packet's arrival and re-transmission in 
each network node, and updating the information field in each network node for each packet by adding the delay 
occurring in ach nod , i.e., r -transmission time - arrival time. 
5 [0010] U.S. Patent No. 4,530,091 , for Synchronization of reai-time clocks in a packet switching system, to Crockett, 
granted July 1 6, 1 985, describes a means to synchronize a remote real-time clock to a standard real-time clock via an 
X.25 (CCITT) network, but does not teach or suggest how to synchronize a multimedia stream, and does not consider 
network delay caused in a packet network. 

[0011] U.S. Patent No. 4,918,687 for Digital packet switching networks, to Bustini et al, granted April 17, 1990, 
10 describes a method to avoid consecutive packet loss by clipping voice packets whose contents indicate whether the 
packet is eligible for clipping, if required, in a node queue within the network. 

[001 2] U.S. Patent No. 5,432,790 for Method for allocating internodal link bandwidth in a packet oriented communi- 
cation network to guarantee delay quality-of -service, to Hluchyj et a/., granted July 11 , 1995, describes a method to 
allocate bandwidth on internodal links such that worst case maximum delays and worst case average delays are guar- 
15 anteed while providing statistical gains in maintaining predetermined end-to-end delay QoS. 

[0013] U.S. Patent No. 5,444,709 for Protocol for transporting real time data, to Riddle, granted August 22, 1 995, 

not timely received. 

[0014] U.S. Patent No. 5,450,394 for Delay monitoring of telecommunication networks, to Gruber et a/., granted 
20 September 12, 1995, describes a means to measure a round-trip delay of network transit time, however, the delay 
which must be known for multimedia transmission and synchronization is a one-way network delay. 
[0015] U.S. Patent No. 5,557,724 for User interface, method, and apparatus selecting and playing channels having 
video, audio, and/or text streams, to Sampat etal., granted September 17, 1996, describes a user interface in a com- 
puter system to playback multimedia streams. 
25 [0016] U.S. Patent No. 5,594,660 for Programmable audio-video synchronization method and apparatus for mufti- 
media systems, to Sung etal., granted January 14, 1997, describes a means to playback an audio and video stream 
synchronously to a single network-based receiver. 

[0017] U.S. Patent No. 5,598,352 for Method and apparatus for audio and video synchronizing in MPEG playback 
systems, to Rosenau et a/., granted January 28, 1997, describes a means to playback an audio and video stream 

30 synchronously to a single network-based receiver. 

[0018] U.S. Patent No. 5,623,483 for Synchronization system for networked multimedia streams, to Agrawal etaf., 
granted April 22, 1 997, describes a means for a receiving device to determine the total end-to-end delay from a statistical 
network delay distribution and the synchronization of the two media streams from different sources through the provision 
of a synchronization clock located in each server and in each receiver. 

35 [0019] U.S. Patent No. 5,629,936for Control of consecutive packet loss in a packet buffer, to Lee etal,, granted May 
13, 1997, describes a method for dropping packets to control consecutive packet loss based on a desired packet gap. 
[0020] U.S. Patent No. 5,675,573 for Delay-minimizing system with guaranteed bandwidth delivery for real-time traf- 
fic, to Karol era/., granted October 7, 1997, describes a bandwidth allocation mechanism based on individual guaran- 
teed bandwidth requirements associated with each traffic flow. 

40 [0021] U.S. Patent No. 5,682,384 tor Apparatus and methods achieving multiparty synchronization for real-time net- 
work application, to Zarros, granted October 28, 1997, describes a means to playback contents from multiple sources 
synchronously to a single receiver on a network. 

[0022] U.S. Patent No. 5,768,527 for Device, system and method of real-time multimedia streaming, to Zhu et al., 
granted June 16, 1998, describes a method to recover lost packets by retransmission. 
45 [0023] U.S. Patent No. 5,802,060 for Data network and a process for real time transmission of data packets in a data 
network, to Fischbach et al., granted September 1, 1998, describes a method to prioritize packet transmissions for 
stations whose network access is controlled by token exchanges. 

[0024] U.S. Patent No. 5,81 5,634 for Stream synchronization method and apparatus for MPEG playback system, to 
Daum etal., granted September 29, 1998, describes a means to playback an audio and video stream synchronously 

50 to a single network-based receiver. 

[0025] U.S. Patent No. 5,844,600 for Methods, apparatus, and systems for transporting multimedia conference data 
streams through a transport network, to Kerr, granted December 1 , 1998, describes a means for audio/video confer- 
encing over an ATM network. Audio streams are combined in a network-based mixer and distributed to terminals. At 
each terminal, th received audio and video str ams are played synchronously, however, there is no provision for 

55 synchronous play at multiple terminals on a network. 

[0026] U.S. Patent No. 5,874,997 for Measuring and regulating synchronization of merged video and audio data, to 
Haigh, granted February 23, 1999, describes a means to playback an audio and video stream synchronously to a 
single, network-based receiver. 
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[0027] The Internet Engineering Task Fore (IETF) publish s requests for comments (RFC) and comments filed In 

respons thereto. 

[0028] RFC 1 889: A real-time end-to-end protocol utilizing existing transport layers for data that has real-time prop- 
erti s. 

[0029] RFC 2508: Compressing IP/UDP/RTP Headers for Low-Speed Serial Links 

[0030] RFC 1144: Compressing TCP/IP Headers for Low-Speed Serial Links, 

[0031] Similar to RFC 2508, but for TCP/IP packets. 

Summary of the Invention 



[0032] A system for transmitting and synchronizing real-time multimedia content includes a multimedia server for 
generating a multimedia packet having multimedia audio/visual information therein in a play sequence, and for trans- 
mitting the multimedia packet; a packet-based communication network connected to the multimedia server for receiving 
the multimedia packets therefrom; the network having multiple routers therein to route the multimedia packets to plural 

15 destinations; means for inserting total delay information (TDI) into the multimedia packets, wherein the TDI includes 
total end-to-end delay (TED) and cumulative network delay (CND); and plural multimedia receivers for receiving the 
multimedia packets having l Ul therein, wherein eacn multimedia receiver includes a buffer for storing received multi-" 
media packets, a sequencing means for retrieving stored multimedia packets from the buffer and sequencing the 
multimedia packets in the play sequence, and a playing mechanism for playing the retrieved, sequenced multimedia 

20 packets at timeTED after the multimedia packet has been transmitted by the multimedia server. A method of transmitting 
and synchronizing real-time multimedia content includes loading multimedia audio/visual into plural multimedia pack- 
ets; inserting total delay information (TDI) into at least one of the multimedia packets, wherein TDI includes total end- 
to-end delay (TED) and cumulative network delay (CND); transmitting the multimedia packets to plural multimedia 
receivers over a packet-based network in a play sequence; buffering the multimedia packets in each multimedia re- 

25 ceiver, recalling the multimedia packets in play sequence; and playing the multimedia packets at each receiver at time 
TED. 

[0033] An object of the invention is to provide synchronized A/V output at a plurality of receivers over and multimedia 
carrying network. 

[0034] Another object of the invention is to prevent transmissions of any multimedia packets which will not be used 
30 by a receiver on the network. 

[0035] These and other objects and advantages of the invention will become more fully apparent as the description 
which follows is read in conjunction with the drawings. 
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Brief Description of the Drawings 



[0036] Fig. 1 depicts a prior art multimedia content delivery system. 

[0037] Fig. 2 depicts a packet-based real-time multimedia content transmission system. 

[0038] Fig. 3 is a block diagram of packet receipt and playback. 

[0039] Fig. 4 depicts a LAN multimedia content delivery system constructed according to the invention. 

40 [0040] Fig. 5 depicts a multimedia content delivery system wherein the video and audio streams are separately 
routed. 

Detailed Description of the Preferred Embodiment 



45 [0041] Synchronization and transmission of multicast multimedia, particularly over packet-based systems is an effi- 
cient and inexpensive techniques for distributing information. Prior art systems simply do not consider the problems 
associated with synchronizing and transmitting a multimedia data^stream, from one or more sources, over a packet- 
based system, to multiple receivers. Although there exist a number of technologies that enable a plurality of correlated 
media streams, transmitted from different locations, to be played at single receiver in a synchronized manner, the 

so known prior art does not allow media streams generated by a single source, or multiple sources, to be multicast over 
a packet-based network to be played synchronously at multiple locations. This invention provides a system and method 
to transmit and synchronously play the same audio/video stream at the different locations, and to eliminate transmission 
of a mul^jmedia packet when it is apparent that the multimedia packet will not be played at its destination. 
[0042] R ferring now to Fig. 4, a content synchroniz d multimedia transmission syst m constructed according to 

55 the invention is depicted generally at 60. System 60 includes a multimedia server 62, which has the means to insert 
TED and CND information, which is defined and described later herein, into each multimedia packet 64 that passes 
ther through. A packet network 66 includes two routers, 68, 70, although many more routers are likely present in such 
a generalized network. Router 68 is connected to a local area network (LAN) 72, which includes a gateway 74 and 
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routers 76, 78. Multimedia r ceivers 80, 82 and 84 ar connecter to LAN 72. 

[0043] The method of th invention includes loading multimedia information into a series of packets 64, referred to 
herein as multimedia packets. Multimedia server 62 inserts fields of total end-to-end delay (TED) and cumulative net- 
work delay (CND) in each multimedia packet, r ferred to herein as total d lay information (TDI). TED indicates the 
time from when th multim dia packet is transmitted from the server to when the multimedia packet is to be played at 
a receiver, whereas CND indicates the actual latency caused by routers in the communications network before the 
multimedia packet is delivered to a receiver. Multimedia server 62 determines the value of TED from network statistics. 
When transmitting a multimedia packet, server 62 sets the value of jitter to CND. The jitter is obtained by 

time interval of transmissions of two consequent packets - packet rate. (1) 

For example, if the packet rate is 20 ms and a packet is actually transmitted 30 ms after the previous packet, jitter is 
1 0 ms and the CND of the packet is set to 1 0 ms. 

[0044] Each router in the network, regardless of whether the router is located in packet network 66 or LAN 72, 
measures the queuing delay of each multimedia packet caused by that router. The queuing delay is obtained from time 
-stamps-o f r e c e pt i o n and transmis s ion -of^ten^mimn^a 
updates CND by adding the queuing delay to the current CND value. If the router needs to route a received multimedia 
packet to multiple routes, i.e., multicast, such as router 68 multicasting to router 70 and gateway 74, the multimedia 
packet is duplicated, and the queuing delay for the duplicated multimedia packet is handled separately from the original. 
There may be more than one duplicated packet, and the queuing delay is placed in each duplicated packet. 
[0045] Each multimedia receiver includes a buffer "B" to store incoming multimedia packets, and recalls a multimedia 
packet from the buffer to play the content of a multimedia packet at (TED-CND) after its reception. Thus, a multimedia 
packet delivered to the multiple multimedia receivers is played at TED after transmitted at the server, regardless of the 
actual network delay caused in the route to each receiver. 

[0046] As described in the previous example, CND contains only a queuing delay, and does not include physical 
latencies caused by the physical transmission between two routers. The physical latency is assumed to be small and 
is therefore negligible. Alternatively, a router may be provided with a known physical latency, which may be added to 
the queuing delay for increased accuracy. 

[0047] If all the receivers are aware of the packet rate, performing the procedures described above for the first 
multimedia packet of the media stream is sufficient, because a receiver receiving a multimedia packet wfth TED and 
CND knows when to play the multimedia packets which follow, however, where the first multimedia packet containing 
TED and CND is "lost," i.e., not received by one or more receivers, the server may be required to include TED and 
CND in the first several multimedia packets. Alternatively, the server may be required to include TED and CND until it 
r ceives an acknowledgement that a multimedia packet having TED and CND has been received by all of the receivers 
belonging to the multicast group. 

[0048] As mentioned earlier, multimedia server 62 determines the value of TED, based on network statistics. TED 
may be provided to the server by a network administrator, or measured dynamically. The actual determination of TED 
is beyond of the scope of this invention, however, U.S. Patent No. 5,623,483 provides a technique to determine TED 
from the statistical distribution of network delay. The overall solution of U.S. Patent No. 5,623,483, however, is that the 
server and the receivers have synchronized clocks. The server specifies, in each packet, the time at which the receiver 
is to play the packet content. However, it is unrealistic to expect that the servers and receivers of a packet network 
maintain precisely the same time on their synchronized clocks. 

[0049] It is foreseeable that situations may arise where queuing delays and physical latencies add up to the point 
where a multimedia packet reaches a receiver, and is "late." In the case of prior art systems, audible and visual dis- 
continuities are perceptible to a user, without any means to correct this situation. The system and method of the in- 
vention have provisions to function in the case where one of the receivers detects a CND in a multimedia packet which 
xceeds the TED for the multimedia packet. In this case, the multimedia packet cannot be played synchronously, 
because the time for play of the contents of the multimedia packet has already passed. The receiver includes means 
to request a larger TDI, and sends a message to the server with the value CND-TED. The server then increases the 
value of TED, and thus TDI, for following multimedia packets. 

[0050] In the embodiment shown in Fig. 4, server 62 and all the routers included in packet network 66 and LAN 72 
for the multicast group supports TED and CND. As depicted in Fig. 4, all of the multimedia receivers in the multicast 
group are in a local area network (LAN), however, a multimedia receiver may also be connect d to a router in packet 
n twork 66. A multim dia server does not have to include special hardware or software to insert TDI into the multimedia 
pack ts, as d scribed above. Instead, a gat way, which connects the LAN to the (public) packet network, generates 
TDI from TED and CND for multimedia packets received from the server. The gateway must be capable of generating 
the value of TED, and must be supplied with th packet rate. The routers and th receivers in the LAN may perform 
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the same function. Consequently, the invention may be implemented ev n if the entire packet network does not support 
the m thod presented here. Put another way, the means for inserting TDI may be provided by a multimedia server, a 
router, or a gateway. 

[0051] Another embodiment, illustrated in Fig. 5, generally at 90, Includes a multim dia server 92, which generates 

5 a video stream 94 and an audio stream 96, a packet network 98, having routers 100, 102, 104, 106 and 108, and a 
video monitor 1 1 0 and a speaker 1 1 2. Video stream 94 and audio stream 96 are to be played separately yet synchro- 
nously at video monitor 1 1 0 and at speaker 112, respectively, which are connected to packet network 98. The destination 
addresses of video packets and audio packets are the addresses for the video monitor and for the speaker, respectively. 
Server 92 includes means to insert TDI , including TED and CND, in both video and audio packets. The routers perform 

10 the same procedure as shown in the other embodiments. 

[0052] The method of U.S. Patent No. 4,506,358 is different from the invention described herein In that the present 
invention employs an information field in each packet of a type neither taught nor suggested in the prior art, that indicates 
the time when the audio/video content in the packet should be played. This enables the same content to be played 
synchronously at multiple receivers, without the necessity of synchronized server/receiver clocks. 

15 [0053] This invention also maximizes the efficiency of bandwidth on physical links among routers and receivers by 
preventing a router from transmitting multimedia packets that will not be played at the final destination, i.e., a media 

player, or receiver, as In the case where CND > TED. 

[0054] Non-viable multimedia packets may be discarded if it is determined that they are not need, or if it is determined 
that they will not be used. The following may be used as a stand-alone technique, or in conjunction with the synchro- 

20 nization technique previously described herein. A router places a time stamp and sequence number in every received 
multimedia packet transferred between the media source and the receiver. Let: 

f Jn]: the time stamp of the multimedia packet with sequence number n, and 

fp[n]: the timing information included in the multimedia packet with sequence number n, indicating when the 
25 content in this multimedia packet is to be played at the receiver. This Information provides relative timing 

information among series of multimedia packets and does not indicate the absolute time. 

[0055] At the receipt of the first multimedia packet, having sequence number = the router stores ynj and y/ij 
in memory and re-directs the multimedia packet to the next router or the receiver Now, let 
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40 



45 



T u = tJ[n 0 ] and (2) 

T p =t p [n 0 ). (3) 

[0056] At the receipt of every following received multimedia packet with sequence number n, the router examines 
the values of yn] and t^n], and, if 

t ts [n]-T ts <tpln]-T p +D, (4) 

where D denotes the maximum TED allowance, including the buffering delay in the receiver, the router re-directs the 
multimedia packet and sets 

T te = r te [n] and (5) 



T p =t p [n]; (6) 

and determines that the multimedia packet is a viable packet. Otherwise, the router discards the multimedia packet. 
[0057] The value t^n]- T te is the difference between the arrival time of the multimedia packet of interest and the 
lat st succ ssf ully re-directed multimedia packet at th router. Similarly, t^n]- T p indicates the time interval at play back 
between these two multimedia packets. Thus, if 
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y^l " T te > ' P [n J -T p +D, (7) 

the multimedia packet with sequence number n has already consumed the entire TED allowance. In this case, there 
is no chance for the multimedia packet to be played at the receiver, because the multimedia packet will not arrive at 
the receiver before or at the time it is to be played. By eliminating unnecessary transmission of such a multimedia 
packet, referred to herein as a non-viable packet, the bandwidth on the physical link between this router and the next 
entity, regardless of whether the next entity is a router or a receiver, may be utilized efficiently. The router thus includes 
means to determine whether a multimedia packet is a viable multimedia packet, in which case the multimedia packet 
is transmitted; or whether the multimedia packet is a non-viable multimedia packet, in which case the multimedia packet 
is discarded. 

[0058] The value of D at any router may be provided by: (1 ) being hard-coded on the node's software or hardware; 
(2) being manually configured by a system administrator; or (3) being provided in every real-time multimedia packet 
by the multimedia server. The value is determined by the required QoS and must be greater than the buffering delay 
in the receiver. D will be a small value for delay-sensitfve applications, such as two-way voice conversation. The receiver 
may negotiate the value of D with the media source using common communication protocols. 

[° 059 J The transmisste^riTretrw^^ wi th I P/UDP/RT P h eader co m presso r , speclf leiMn-nFC ' 

2508, wherein the header information of real-time multimedia packets is compressed by a differential coding scheme. 
The compressor, which may be implemented in a transmitting portion of a router, provides a means to identify a specific 
real-time media stream between a media source and a receiver, among other traffic. Because of differential coding, 
the compressor is required to examine the timing information in the header of real-time multimedia packets in any 
packet-based network, and the extra processing power required to implement the method of the invention is minimal. 
[0060] Thus, a system and method for transmitting and synchronizing real-time, multimedia packet-based content 
over a packet-based network has been disclosed. Although several variations of the system and method of the invention 
have been disclosed, it will be appreciated that further variations and modification may be made thereto without de- 
parting from the scope of the invention as defined in the appended claims. 

Claims 

1. A system (60) for transmitting and synchronizing real-time multimedia content, comprising: 

a multimedia server (62) for generating a multimedia packet (64) having multimedia audio/visual information 
therein in a play sequence, and for transmitting said multimedia packet (64); 

a packet-based communication network (66) connected to said multimedia server (62) for receiving said mul- 
timedia packets therefrom; said network having multiple routers (68, 70) therein to route said multimedia pack- 
ets (64) to plural destinations; 

means for inserting total delay information (TDI) into said multimedia packets (64), wherein said TDI includes 
total end-to-end delay (TED) and cumulative network delay (CND); and 

plural multimedia receivers (80, 82, 84) for receiving said multimedia packets (64) having TDI therein, wherein 
each multimedia receiver (80, 82, 84) includes a buffer for storing received multimedia packets, a sequencing 
means for retrieving stored multimedia packets from said buffer and sequencing said multimedia packets in 
said play sequence, and a playing mechanism for playing the retrieved, sequenced multimedia packets at time 
TED after said multimedia packet (64) has been transmitted by said multimedia server. 

2. The system (60) of claim 1 wherein said means for inserting is located in said multimedia server (62). 

3. The system (60) of claim 1 wherein said means for inserting is located in a router (68, 70). 

4. The system (60) of claim 1 which further includes a local area network (LAN) (72) located between said packet- 
based network (66) and at least some of said multimedia receivers (80, 82, 84), wherein said LAN (72) includes 
a gateway (74). 

5. The system (60) of claim 4 wherein said means for ins rting is located in said gateway (74). 

6. The system (60) of claim 1 wherein at least one of said routers (68, 70) includes duplication means for duplicating 
multimedia packets (64), wherein said means for inserting is located in said router (68, 70), and wherein said router 
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(68, 70) inserts TDI into each duplicated multimedia packet. 

7. The system (60) of claim 1 which further includes means to determine whether a multimedia packet (64) is a viable 
packet or whether a multimedia packet is a non-viable pack t, andwher in said non-viable packets are discarded. 

5 

8. The system (60) of claim 1 wherein said multimedia receivers (80, 82, 84) includes means to request a larger TDI 
from said means for inserting. 

9. A method of transmitting and synchronizing real-time multimedia content, comprising: 

10 

loading multimedia audio/visual into plural multimedia packets (64); 

inserting total delay information (TDI) into at least one of the multimedia packets (64), wherein TDI includes 
total end-to-end delay (TED) and cumulative network delay (CND); 

transmitting the multimedia packets (64) to plural multimedia receivers (80, 82, 84) over a packet-based net- 
15 work (66) in a play sequence; 

buffering the multimedia packets (64) in each multimedia receiver (80, 82, 84); 

r eca l l i ng th e m ult imedi a packets (64) i n p l ay sequ e nc ej-end 

playing the multimedia packets (64) at each receiver (80, 82, 84) at time TED. 

20 10. The method of claim 9 wherein said inserting includes inserting TDI in a multimedia server (62). 

1 1 . The method of claim 9 wherein said inserting includes inserting TDI in a router (68, 70) located in the packer-based 
network (66). 

25 12. The method of claim 9 wherein said inserting includes Inserting TDI in a gateway (74) located in a local area 
network (LAN) (72) which is located between the packet-based network (66) and a multimedia receiver (80, 82, 84). 

13. The method of claim 9 which includes duplicating a multimedia packet (64) for multicasting to plural multimedia 
receivers (80, 82, 84), and wherein said duplicating includes inserting TDI into at some of the duplicated multimedia 

30 packets. 

14. The method of claim 9 which includes discarding a multimedia packet (64) if at any time during said transmitting 
CND>TED. 

35 15. The method of claim 14 which includes increasing the value of TED if CND>TED. 

16. The method of claim 9 which includes determining whether a multimedia packet (64) is a viable packet or a non- 
viable packet, and which further include discarding any non-viable packets. 

40 17. A method of transmitting and synchronizing real-time multimedia content, comprising: 

determining whether a multimedia packet (64) is a viable packet or a non-viable packet; and 
discarding any non-viable packets. 

45 18. The method of claim 1 7 which further includes: 

loading multimedia audio/visual into plural multimedia packets (64); 

inserting total delay information (TDI) into at least one of the multimedia packets (64), wherein TDI includes 
total end-to-end delay (TED) and cumulative network delay (CND); 
so transmitting the multimedia packets (64) to plural multimedia receivers (80, 82, 84) over a packet-based net- 

work (66) in a play sequence; 

discarding a multimedia packet (64) if at any time during said transmitting CND>TED; 
buffering the multimedia packets (64) in each multimedia receiver (80, 82, 84); 
recalling the multim dia packets (64) in play sequence; and 
55 playing th multimedia packets (64) at each receiver (80, 82, 84) at time TED. 

19. The method of claim 18 wherein said inserting includes inserting TDI in a multimedia server (62). 
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20. The method of claim 18 wher in said inserting includes inserting TDI in a router (68, 70) located in the packet- 
based network (66). 

21. The method of claim 18 wherein said ins rting includes inserting TDI in a gateway (74) located in a local area 
5 network (LAN) (72) which is located between the packet-based network (66) and a multimedia receiver (80, 82, 84). 

22. The method of claim 1 8 which includes duplicating a multimedia packet (64) for multicasting to plural multimedia 
receivers (80, 82, 84), and wherein said duplicating includes inserting TDI into at some of the duplicated multimedia 
packets. 

10 

23. The method of claim 1 8 which includes increasing the value of TED if CND>TED. 
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